Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added .NET client for Dapr Jobs API #1331

Open
wants to merge 52 commits into
base: master
Choose a base branch
from

Conversation

WhitWaldo
Copy link
Contributor

Description

Added a Dapr Jobs client for the new Jobs API.

This PR is intended to replace the previous one and was performed as a squash merge of it - the third commit (early on) was missing a period in my email address which apparently breaks the DCO process (even though I have an email alias and I can still receive email there). Because I cannot figure out how to fix multi-line comments, I'm providing this updated PR instead.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #1321

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation
  • Added sample project

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
@WhitWaldo
Copy link
Contributor Author

@philliphoff Any other changes you think should be made here?

@WhitWaldo
Copy link
Contributor Author

I'm seeing the E2E tests fail on my system following the recent merge with master. In hopes those are due to something wrong with those tests, I'll hold off on any other changes to this PR to address them until it's confirmed it's a problem with this PR and not them.

I did add two more unit tests to validate the DaprClientBuilder a bit more where it was lacking existing tests and those are passing.

@yaron2
Copy link
Member

yaron2 commented Aug 21, 2024

cc @halspang @philliphoff

/// runtime gRPC client used by the consuming package.
/// </summary>
/// <exception cref="InvalidOperationException"></exception>
protected (GrpcChannel channel, HttpClient httpClient, Uri httpEndpoint) BuildDaprClientDependencies()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this tuple will become unwieldy if/when the number of dependencies/options increase. I wonder if something like this would be more flexible:

public TClientBuilder Build()
{
  DaprClientDependencies dependencies = // Generate dependencies;

  return this.Build(dependencies);
}

protected virtual TClientBuilder Build(DaprClientDependencies dependencies);

protected sealed record DaprClientDependencies(GrpcChannel Channel, HttpClient HttpClient, Uri HttpEndpoint);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pulled this out primarily to remove duplicate code when spinning up clients in other packages ensuring that they're all using the same configuration data for endpoints, API keys and whatnot.

However, as was evidenced in the subscriptions implementation, this approach doesn't really lend itself to any use of injected functionality down the line (e.g. logging, compression handlers, serialization handlers, etc) provided by the developer at registration time. Do you have any thoughts on abandoning this approach altogether and instead opting for more of a DI-first, factory-style approach?

I'd be happy to put together a POC in a separate PR if you're more inclined to visit that elsewhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the similarity of the downloads between Dapr.Client and Dapr.AspNetCore, I'm curious if you know if there have been any surveys done on just how many people use the DaprClient without relying on dependency injection?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know of any, though my personal expectation is that more people are using the AspNetCore provided DI method than building one manually.

…ions.Configuration to InternalsVisibleTo on DaprClient. Whole solution builds without error now locally.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
@WhitWaldo
Copy link
Contributor Author

@philliphoff I've thought about where you were coming from in the review and have overhauled the API substantially. There's now a single ScheduleJobAsync method that takes a DaprJobSchedule along with the other optional arguments. Each of the scheduling extension methods have been removed in favor of two that take an object and serialize it with the provided (albeit optional) JsonSerializerOptions and another that takes a string (presumably already serialized) and simply encodes it to UTF-8 bytes.

That leaves the GetJobsAsync and DeleteJobsAsync otherwise untouched.

A DaprJobSchedule supports the prefixed periods, Cron expressions, duration intervals and point-in-time date/time values via a series of factory methods:

  • Yearly or Monthly (among others, except "@every") for prefixed "@Yearly" and "@monthly" expressions, respectively)
  • FromDateTime accepts a DateTimeOffset with a specified point-in-time value and converts to an ISO8601 value
  • FromDuration accepts a TimeSpan and converts to the Goland duration value
  • FromExpression accepts an open-ended string expression (and the comment calls out that it accepts either a Cron-like or '@' prefixed period string - it doesn't do any validation except null/empty checking
  • FromCronExpression is for those that want a more guided experience building Cron expressions. I added a CronExpressionBuilder class with a fluent API for building out various valid expressions (on T value, every U period, from V to W periods, on X,Y,Z days/months). It overloads ToString to output the resulting expression, but it can be passed directly into FromCronExpression if the user wants to take that approach over providing their own value to FromExpression.

The DaprJobSchedule class uses an internal constructor - it's expected the users will use the factory methods to build it (this was mostly to avoid just having umpteen constructor overloads, but if you think that approach is clearer over the factories, I can pivot).

Because a JobDetails now returns a DaprJobSchedule, the former also includes four boolean properties identifying the expression value as a prefixed period, point-in-time, interval or Cron expression and the value itself can be retrieved from ExpressionValue.

I removed the exceptions as they weren't actually used anymore and cleaned up the larger solution to remove the use of the two shared types in favor of accessing them from Dapr.Common. I've set up InternalsVisibleTo for each project in the solution in that project so that the types aren't exposed as part of the public API, so this PR now tweaks just about all projects in the solution in this regard.

I'm working to add more unit tests for the CronExpressionBuilder and make sure I didn't miss any edge situations.

I'm going to go back through the remaining review comments and address each, but I wanted to put the latest changes in one easy-to-find place.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…vocation

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
… bugs along the way)

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…age.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
@WhitWaldo
Copy link
Contributor Author

WhitWaldo commented Sep 15, 2024

In the last several commits, I've been working to build out a regular expression that can evaluate whether a value in a DaprJobSchedule is a Cron expression or not (so it can be a bit more reliable than "it's not any of the other types, so it must be Cron"). I've made incremental progress to capture each of the separate segments, add testing and address unsupported cases, but it passes the current test suite which is pretty comprehensive right now.

I did want to call out why I opted for the approach I did in the latest commit. I created a separate project to run some benchmarking because I wanted to implement this in a performant way that also limited allocations and the results were mildly surprising. I have three different approaches called SeparateRegex, CombinedRegex and GeneratorRegex and they differ as follows:

SeparateRegex

This approach reflects how I developed the whole expression - one segment at a time. It works by splitting the expression by a whitespace character. If there are fewer than 6 segments, it's not a Cron expression. Otherwise, evaluate each segment by its respective regular expression and return a boolean value reflecting if they all were a match or not.

CombinedRegex

This approach builds off the completed work of the last one in that I create a compiled regular expression of the entire string (as follows) and evaluate the expression as provided:

([0-5]?\d-[0-5]?\d)|([0-5]?\d,?)|(*(/[0-5]?\d)?) ([0-5]?\d-[0-5]?\d)|([0-5]?\d,?)|(*(/[0-5]?\d)?) (([0-1]?\d)|(2[0-3])-([0-1]?\d)|(2[0-3]))|(([0-1]?\d)|(2[0-3]),?)|(*(/([0-1]?\d)|(2[0-3]))?) *|(*/(([0-2]?\d)|(3[0-1])))|(((([0-2]?\d)|(3[0-1]))(-(([0-2]?\d)|(3[0-1])))?)) (^(*/)?((0?\d)|(1[0-2]))$)|(^*$)|(^((0?\d)|(1[0-2]))(-((0?\d)|(1[0-2]))?)$)|(^(?:JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)(?:-(?:JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC))?(?:,(?:JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)(?:-(?:JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC))?)*$) *|(*/(0?[0-6])|(0?0-6?)|((,?(SUN|MON|TUE|WED|THU|FRI|SAT))+)|((SUN|MON|TUE|WED|THU|FRI|SAT)(-(SUN|MON|TUE|WED|THU|FRI|SAT))?))

GeneratorRegex

This utilizes the new approach introduced in .NET 7 of using a source generator to produce a pared down version of the regular expression engine to only those bits necessary for the provided expression.

The results of a standard benchmarking run + memory diagnostics are as follows:
image

Surprisingly, Separate performed the best, followed by the Generator approach and then Combined (all within 2 microseconds of one another, so pretty similar). But the allocations were a distinctly different story: Separate performed the worst by far with 5616 bytes versus 216 bytes for Combined and 152 for the Generator, which make a lot of sense and align with expectations.

As the 1.15 release of Dapr will precede the release of .NET 9 and will continue to have full support for .NET 6, it means that while Regex source generators were introduced with .NET 7, use of the GeneratedRegexAttribute itself requires .NET 7 and and later so it's not yet available for all targeted versions (Dapr targets the maximum common language version of C# 10 right now) so in a future release, we can make this tweak and save a few more kb of allocations.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
/// </summary>
/// <param name="value">Argument value to check.</param>
/// <param name="name">Name of Argument.</param>
public static void ThrowIfNullOrEmpty([NotNull] string? value, string name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: ArgumentException.ThrowIfNullOrEmpty() was added in .NET 7 (so we should be able to ditch this these methods as soon as .NET 6 is deprecated.

…fNullOrEmpty in Jobs SDK implementation

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…mparisons. Updated all usages.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…ng with each call.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Copy link

codecov bot commented Sep 26, 2024

Codecov Report

Attention: Patch coverage is 77.77778% with 84 lines in your changes missing coverage. Please review.

Project coverage is 68.01%. Comparing base (1b7c9f4) to head (d6e88d2).
Report is 19 commits behind head on master.

Files with missing lines Patch % Lines
src/Dapr.Jobs/DaprJobsGrpcClient.cs 33.33% 45 Missing and 7 partials ⚠️
...apr.Jobs/Extensions/DaprSerializationExtensions.cs 0.00% 11 Missing ⚠️
src/Dapr.Common/DaprDefaults.cs 66.66% 4 Missing and 6 partials ⚠️
src/Dapr.Jobs/DaprJobsClient.cs 37.50% 5 Missing ⚠️
....Jobs/Extensions/EndpointRouteBuilderExtensions.cs 0.00% 4 Missing ⚠️
.../Extensions/DaprJobsServiceCollectionExtensions.cs 92.85% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1331      +/-   ##
==========================================
+ Coverage   67.28%   68.01%   +0.72%     
==========================================
  Files         174      187      +13     
  Lines        6025     6352     +327     
  Branches      671      722      +51     
==========================================
+ Hits         4054     4320     +266     
- Misses       1802     1853      +51     
- Partials      169      179      +10     
Flag Coverage Δ
net6 67.98% <77.77%> (+0.72%) ⬆️
net7 67.98% <77.77%> (+0.72%) ⬆️
net8 67.99% <77.77%> (+0.72%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…int failure when a cancellation was requested

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Jobs API needs a .NET client
3 participants